On Agnostic Learning of Parities, Monomials, and Halfspaces

We study the learnability of several fundamental concept classes in the agnostic learning framework of Haussler [Hau92] and Kearns et al. [KSS94]. We show that under the uniform distribution, agnostically learning parities reduces to learning parities with random classification noise, commonly referred to as the noisy parity problem. Together with the parity learning algorithm of Blum et al. [BKW03], this gives the first nontrivial algorithm for agnostic learning of parities. We use similar techniques to reduce learning of two other fundamental concept classes under the uniform distribution to learning of noisy parities. Namely, we show that learning of DNF expressions reduces to learning noisy parities of just logarithmic number of variables and learning of k-juntas reduces to learning noisy parities of k variables. We give essentially optimal hardness results for agnostic learning of monomials over {0, 1}n and halfspaces over Q. We show that for any constant 2 finding a monomial (halfspace) that agrees with an unknown function on 1/2 + 2 fraction of examples is NP-hard even when there exists a monomial (halfspace) that agrees with the unknown function on 1− 2 fraction of examples. This resolves an open question due to Blum and significantly improves on a number of previous hardness results for these problems. We extend these results to 2 = 2− log 1−λ n (2 = 2− √ log n in the case of halfspaces) for any constant λ > 0 under stronger complexity assumptions. Preliminary versions of the results in this work appeared in [Fel06] and [FGKP06]. ∗Work done while the author was at Harvard University supported by grants from the National Science Foundation NSF-CCR0310882, NSF-CCF-0432037, and NSF-CCF-0427129. †Work done while the author was at Georgia Tech. ‡Supported in part by Subhash Khot’s Microsoft New Faculty Fellowship and Raytheon Fellowship, College of Computing, Georgia Tech.